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Abstract 

The possibility to determine the type of neutrino mass hierarchy by studying atmo- 
spheric neutrino oscihations with a detector capable to distinguish between neutrino 
and antineutrino events, such as magnetized iron calorimeters, is considered. We dis- 
cuss how the ability to distinguish between the neutrino mass spectrum with normal 
and inverted hierarchy depends on detector characteristics like neutrino energy and 
direction resolutions or charge miss-identification, and on the systematical uncertain- 
ties related to the atmospheric neutrino fluxes. We show also how the neutrino mass 
hierarchy determination depends on the true values of ^13 and ^23) as well as on the 
type of the true hierarchy. We find that for /i-like events, an accurate reconstruction of 
the energy and direction of the neutrino greatly improves the sensitivity to the type of 
neutrino mass spectrum. For sin^ 2^13 = 0.1 and a precision of 5% in the reconstruction 
of the neutrino energy and 5° in the neutrino direction, the type of neutrino mass hier- 
archy can be identified at the 2a C.L. with approximately 200 events. For resolutions 
of 15% for the neutrino energy and 15° for the neutrino direction roughly one order 
of magnitude larger event numbers are required. For a detector capable to distinguish 
between and induced events the requirements on energy and direction resolutions 
are, in general, less demanding than for a detector with muon charge identification. 
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1 Introduction 



There has been a remarkable progress in the studies of neutrino oscillations in the last 
several years. The experiments with solar, atmospheric and reactor neutrinos [1-5] have pro- 
vided compelling evidences for existence of neutrino oscillations driven by nonzero neutrino 
masses and neutrino mixing. Evidences for oscillations of neutrinos were obtained also in the 
first long-baselinc accelerator neutrino experiment K2K [6]. The interpretation of the solar 
and atmospheric neutrino, and of KamLAND data in terms of neutrino oscillations requires 
the presence of 3-neutrino mixing in the weak charged lepton current (see, e.g., [7]): 

3 

^iL^^UijUji^, l^e,fx,T. (1) 

Here, uil are the three left-handed flavor neutrino fields, uji^ is the left-handed field of the 
neutrino uj with the mass ruj and U is the Pontecorvo-Maki-Nakagawa-Sakata (PMNS) 
neutrino mixing matrix [8], 

(Uel Ue2 Ue3 \ / C12C13 S12C13 8136''^ \ 

U^l U^,2 U^2. = j -S12C23 - Ci2S23Sl3e** C12C23 " Sl2S23Sl3e*'' S23C13 
Url Ur2 UrZ ) \ S12S23 " Ci2C23Sl3e"^ -C12S23 " Sl2C23Sl3e"^ C23C13 / 

(2) 

where wc have used a standard parametrization of U with the usual notations, Sij = sinOij, 
Cij = cos 9ij, and 6 is the Dirac CP- violation phase ^. If one identifies Aml^ > and Am|i 
(or Amga) with the neutrino mass squared differences which drive the solar and atmospheric 
neutrino oscillations, 612 and ^23 represent the solar and atmospheric neutrino mixing angles, 
while ^13 is the angle limited by the data from the CHOOZ and Palo Verde experiments [12]. 
It follows from the data that |Am3^^2)l ^ Am^i. In this case, obviously, Am|i ^ Am32. 

The existing solar and reactor neutrino oscillation data allow us to determine Am^i and 
sin^ 612 with a relatively good precision and to obtain rather stringent limits on sin^ ^^13 (see, 
e.g., [2-5, 13-15]). The best fit values and the 99.73% C.L. allowed ranges of Amli and 
sin^ 012 are given by ^: 

Ami = 8.0 X 10-5 eV^, sin^ 612 = 0.31 , 

(3) 

Amii = (7.1 - 9.0) X 10-5 eV^, sin^ $12 = (0.24 - 0.40) , 

It follows from a combined S-u oscillation analysis of the solar neutrino, KamLAND and 
CHOOZ data that [13, 15] 

sin^ < 0.044 at 99.73% C.L. (4) 

Existing atmospheric neutrino data are essentially insensitive to ^13 obeying the CHOOZ 
limit. The Super-Kamiokande and K2K experimental results are best described in terms of 

^ Wc have not written explicitly the two possible Majorana CP-violation phases [9, 10] which do not enter 
into the expressions for the oscillation probabilities of interest [9, 11]. 

^The data imply, in particular, that maximal solar neutrino mixing is ruled out at 6a. At 95% C.L. 
one finds cos2^0 > 0.26 [13], which has important implications [16]. 
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dominant — > Ur (z/^ — > z/^) vacuum oscillations. The best fit values and the 99.73% C.L. 
allowed ranges of the atmospheric neutrino oscillation parameters read [15]: 

|Am|i| = 2.2 X 10-3 eV^, sin^ ^23 = 0.50 , 

(5) 

I = (1.4 - 3.3) X IQ-^eV^, sin^ 623 = (0.34 - 0.68) . 

The sign of Am|i and of cos2^^23 if sin^ 26'23 < 1-0 cannot be determined using the existing 
data. The two possibilities, Amg^ > or Aml^ < 0, correspond to two different types of 
neutrino mass spectrum: with normal hierarchy (NH), mi < m2 < m^, and with inverted 
hierarchy (IH), < mi < 1712- The fact that the sign of cos2^^23 is not determined when 
sin^ 2^^23 < 1-0 implies that if, e.g., sin^ 26*23 = 0.92, two values of sin^ 6*23 are possible, 
sin^ ^23 = 0.64 or 0.36. 

The neutrino oscillation parameters Am|i, sin^6'i2, |Am|i| and sin^ 6*23 are determined 
by the existing data with a 3a error of approximately 12%, 27%, 50% and 34%, respectively. 
These parameters can (and very likely will) be measured with much higher accuracy in the 
future (see, e.g., [7,15,17]). The MINOS experiment [18], which collects data at present, will 
reduce considerably (approximately by a factor of 3) the current uncertainty in the value 
of |Am3i|. The highest precision in the determination of |Am3i| is expected to be achieved 
from the studies of z/^-oscillations in the T2K experiment with the Super-Kamiokande de- 
tector [19]: if the true |Am|i| = 2 x 10"^ eV^ (and true sin^ 6*23 = 0.5), the 3a uncertainty 
in |Am|i| is estimated to be reduced in this experiment to ~ 12% [17]. The error diminishes 
with increasing of |Am3^|. The T2K experiment will measure also sin^ ^^23 with an error of 
approximately 14% at 2a [15]. However, this experiment would not be able to resolve the 
^23-(7r/2 — ^23) ambiguity if sin^ 2^23 < 1. In what regards the CHOOZ angle ^13, there are 
several proposals for reactor z/g experiments with baseline L ~ (1-2) km [20,21], which could 
improve the current limit by a factor of (5-10). The reactor ^13 experiments can compete 
in sensitivity with accelerator experiments T2K [19], NOz/A [22] (see, e.g., [17]), and can 
be done on a relatively short time scale. The most precise measurement of Am^i could be 
achieved [23] using Super-Kamiokande doped with 0.1% of gadohnium (SK-Gd) for detection 
of reactor Pg [24]: getting the same flux of reactor Pg as KamLAND, the SK-Gd detector will 
have approximately a 43 times bigger reactor z/g event rate than KamLAND. After 3 years 
of data-taking with SK-Gd, l^m^^ could be determined with an error of 3.5% at 3a [23]. A 
dedicated reactor v^. experiment with a baseline L ~ 60 km, tuned to the minimum of the Vf. 
survival probability, could provide the most precise determination of sin^^i2 [25,26]: with 
statistics of ~ 60 GW ktyr and a systematic error of 2% (5%), sin^^i2 could be measured 
with an error of 6% (9%) at 3a [25]. 

Getting more precise information about the value of the mixing angle ^13, and measuring 
the values of l^rn^^, |Am|i|, sin^ ^^12 and sin^ ^23 with a higher precision, is crucial for the 
future progress in the studies of neutrino mixing (see, e.g., [7] and the references quoted 
therein). Here we focus in particular on the question whether the neutrino mass spectrum 
is with normal or inverted hierarchy, i.e., the sign of Am|i, which is of fundamental impor- 
tance for the understanding of the origin of ncTitrino masses and mixing. Information on 
sign(Am|i) can be obtained through matter effects in long-baseline experiments [22,27,28], or 
maybe from supernova neutrinos [29]. If neutrinos with definite mass are Majorana particles, 
information about sgn(Am3i) could be obtained also by measuring the effective Majorana 
mass in neutrinoless double /3-decay experiments [16,30]. 
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In the present article we investigate in detail the possibility to obtain information on the 
sign of Amli using the data on atmospheric neutrinos, which can be obtained in experiments 
with detectors able to measure the charge of the muon produced in the charged current 
reaction by atmospheric or u^. Thus, in contrast to water- Cerenkov detectors, in these 
experiments it will be possible to distinguish between the and induced events. Among 
the operating detectors, MINOS has muon charge identification capabilities for multi-GeV 
muons [18]. The MINOS experiment is currently collecting atmospheric neutrino data. The 
detector has a relatively small mass, but after 5 years of data-taking it is expected to collect 
about 440 atmospheric z/^ and about 260 atmospheric multi-GeV events. The ATLAS 
and CMS experiments under preparation at the LHC accelerator at CERN will have efficient 
muon charge identification for multi-GeV muons and, in principle, can also be used for 
studies of atmospheric neutrino oscillations in the periods when the LHC accelerator will 
not be operational ^. There are also plans to build a 30-50 kton magnetized iron tracking 
calorimeter detector in India within the India-based Neutrino Observatory (INO) project [31]. 
The INO detector will be based on MONOLITH design [32]. The primary goal is to study 
the oscillations of atmospheric and P^. This detector is planned to have efficient muon 
charge identification, high muon energy resolution (~ 5%) and a muon energy threshold of 
about 2 GeV. It will accumulate sufficiently high statistics of atmospheric and z/^ induced 
events in several years, which would permit to perform a high precision study of the z/^ and 

disappearance due to oscillations and to search for effects of the sub-dominant — >■ i/g 
(z/g Uy) and Ue {^e i^fi) transitions. We consider a detector which permits to 

reconstruct the initial (P^) energy and direction with a relatively good precision. The 
magnetized iron detectors can have such capabilities. We present results also for detectors 
permitting to study the oscillations of the atmospheric multi-GeV and Pg under similar 
conditions. 

The possibility to obtain information about the type of neutrino mass spectrum, i.e., 
about the sgn(Am|j^), by investigating atmospheric neutrino oscillations is based on the 
prediction that in the case of 3-neutrino oscillations, the Earth matter affects in a different 
way the transitions of neutrinos and antineutrinos [33-35]. For Am'^i > 0, the z/^ u^, 
(z/g z/^) transitions of the multi-GeV neutrinos are amplified, while the transitions of 
antineutrinos z/^ z/g (z/g v^j) are suppressed. If Amg^ < 0, the indicated transitions 
of neutrinos are suppressed and those of antineutrinos are enhanced. The magnitude of 
the Earth matter effects in the oscillations of multi-GeV atmospheric z/^ (P^) and Ue i^e) 
depends critically on the value of sin^^ia: the effects can be substantial for sin^^ia ^ 0.01; 
for sin^ ^^13 ^ 0.01 they are exceedingly small. The difference between the oscillations of 
neutrinos and antineutrinos can be relatively large and observable in the samples of fi^ 
events with E^^ {2 — 10) GeV, from atmospheric z^'s with a large path length in the Earth, 
crossing deeply the Earth mantle [27] or the mantle and the core [36-40]. These are the 
neutrinos for which cos^^„ ^ (0.3 — 0.4), where On is the Nadir angle characterizing the 
neutrino trajectory in the Earth. 

Let us note that three-flavour effects in atmospheric neutrino oscillations including the 
Earth matter effects have been widely studied in the literature; for an incomplete list see [40- 
55]. A rather detailed analysis for the MONOLITH detector has been performed in [52], and 
magnetized iron detectors in general have been considered previously in [42,53-55]. A large 

^We thank F. Vannucci for pointing out this possibility to us. 
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number of studies have been done for water-Cerenkov detectors [41,47-51]. Such detectors 
have a more hmited sensitivity to the type of neutrino mass hierarchy, because only the sum 
of neutrino and antineutrino induced events can be observed. Since large matter effects occur 
for neutrino if Amg^^ > and for antineutrinos if Amg^ < 0, summing the corresponding 
events significantly diminishes the effect of changing the sign of Am'^i. The remaining 
sensitivity (see e.g., [41,50]) is due to the facts that neutrino and antineutrino cross sections 
differ roughly by a factor of 2, and that a huge number of events is available in mega ton scale 
experiments [48]. Obviously, such a "dilution" of the magnitude of the matter effects does 
not take place in the samples of /i~ and /i^ events, which can be collected in the experiments 
with muon charge identification. 

2 Three-Neutrino Oscillations of Atmospheric Neutrinos 

In the present Section we will review briefly the formalism and the physics of the 3- 
neutrino oscillations of the multi-GeV atmospheric neutrinos with energy E {1 — 10) 
GeV in the Earth (see, e.g., [41] and the references quoted therein for further details). The 
probabilities of the flavour neutrino oscillations in matter of interest, Ua — > I'p, i^a i^p, 
a = e, /i, /3 = e, /i, r, can be written for an arbitrary matter density profile as 

P'.,= \K,\\ (6) 

where A^^ is the probability amplitude which is obtained by solving a Schrodinger-like 
system of coupled evolution equations and p = u {u) for neutrinos (antineutrinos). We use 
the standard parameterization of the PMNS mixing matrix [56], U = O23USO13O12, where 
Oij is the orthogonal matrix of rotations in the ij-plane which depends on the mixing angle 
9ij, and Us = diag(l, l,e'^), 6 being the Dirac CP- violating phase. Exploiting the fact that 
the matrix O23US commutes with the matrix containing the matter potential, it is easy to 
show that [57] without loss of generality A^^ can be written as 

AP = OMA'PUlOl, (7) 

where A'^ does not depend on 623 and S. We adopt further the approximation of setting 
the neutrino mass-squared difference Am^i, responsible for the solar neutrino oscillations, 
to zero In this approximation the probabilities of interest are independent of the mixing 
angle 612 and the Dirac CP-violating phase 6. Moreover, the calculation of A'^ is reduced 
effectively to the calculation of probability amplitudes in the case of two neutrino mixing [57]. 
For neutrinos with energy E and crossing the Earth along a trajectory characterized by a 
Nadir angle 6n, the probability of the 2- neutrino —>■ v' {ve transitions is given by 

P^^ ^ P'^ {Ami, 9,3; E, 9^) = = = 1 - KiP = 1 - lAgW (8) 

where p = u (u), and u' = 8231'^ + C231't [57]. The three-neutrino oscillation probabilities are 

*This can be justified by the fact that we are interested in oscillations of multi-GeV (atmospheric) 
neutrinos in the Earth and that the existing data implies Am|]^/|Arn|]^| = 0.04 ^ 1. We have checked by 
explicit calculations that for the energy range of interest, 1 GeV, our results are not affected by setting 

Am2i = if Am2i has values in the interval given in Eq. 
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obtained as 



p3p ~ 

ee 


|A'fi|2 = 1 - P2P^ 






p3p ~ 


„2 1 A'p |2 _ 2 p2p 
^23 1^13 1 ~ ■'23 ' 






p3p ~ 

er 


^2 1 /i/p |2 _ 2 p2p 
^-23 1^131 "''23-' ' 






p3p ~ 


^23 + '^23 ^3*3 + 2C23 


■^23 






^ ~ ^23 P ^ ^ 2C23S23 


[1- 




p3p — 


1 _ p3p _ p3p 
fifj, fie ■ 







(9) 
(10) 

(11) 

(12) 
(13) 
(14) 

In Eq. dni) we used e-^^^M^, = ^3^3, where and 

known phase and 2-neutrino 

transition probabihty amphtude [36,37,57]. 

The fluxes of atmospheric z/g,^ and i^e^^ of energy E, which reach the detector after crossing 
the Earth along a given trajectory specifled by the value of 9^ are given by the following 
expressions in the case of the 3-neutrino oscillations under discussion [37]: 

^l{E,e^) = [l + (4r^-l)p2p] ^ (15) 

^1{E, On) - <^l{l + 4 [(.^3 r^r' - 1] P'^ - 244 [1 - Re(Ar3)] } , (16) 

where ^^e{fi) ~ '^e(/i)(-^' ^") ^1^^ '^e(M) (P ~ ^) ^^*1 '^e(At) ip = ^) fluxes in the absence of 
neutrino oscillations, and 

^ rP(E 9 ) ^ ^ (17) 

The interpretation of the SK atmospheric neutrino data in terms of dominant z^^ Vr 
and Vr oscillations requires the parameter S23 to lie approximately in the interval 

(0.30 — 0.70), with 0.5 being the statistically preferred value. For the predicted ratio r'^{E, On) 
of the fluxes of atmospheric (z/^) and z/g (z^e) with energy E = {2 — 10) GeV, crossing the 
Earth along trajectories for which 0.3 ^ cos^n < 1.0, one has [58]: r^(£^, 6'„) = (2.6 — 4.5). 
Correspondingly, for S23 = 0.5 (0.64) we flnd for the factors multiplying the 2-neutrino 
transition probability P^p jn Eqs. dEI) and (HSI): (5^3 - 1) = 0.3 - 1.3 (0.66 - 1.9), 
and s^3[l - (s^3 r^)"^] = 0.06 - 0.14 (0.16 - 0.27). This result implies that the 2-neutrino 
transition probability P^^ can have a much larger effect on the flux of atmospheric Ve (j^e) 
than on the flux of atmospheric (^/x)- 

In our further analysis performed in Sections 5 and 6, the matrix A' and correspondingly 
the probabilities P^^ and the amplitudes A!^^ = e'^'^pA^^,, are obtained by solving the rele- 
vant system of evolution equations (see, e.g., [57]) numerically using a realistic Earth density 
proflle provided by the PREM model [59]. As is well-known, the Earth density distribution 
in the existing Earth models is assumed to be spherically symmetric. There are two major 
density structures — the core and the mantle, and a certain number of substructures (shells 
or layers) ^. The mean electron number densities in the mantle and in the core read [59]: 
j^rnan ^ 2.2 A^A cm'^ iV^^ = 5.4 A^A cm'^, A^A being the Avogadro number. 

^According to the Earth models [59,60], the core has a radius Rc — 3485.7 km, the Earth mantle 
depth is approximately Rman = 2885.3 km, and the Earth radius is i?® = 6371 km. The mean values of 
the matter densities and the electron fraction numbers in the mantle and in the core read, respectively: 
Pman = 4.5 g/cm^, = 11.5 g/cm^, and [61] F™"" = 0.49, Y^^ = 0.467. 
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Rather simple analytic expressions for the quantities of interest, P^^, Kp and A^^, have 
been derived in the two- layer model of the Earth density distribution (see, e.g., [56]). In 
this model the electron number densities in the mantle and in the core are assumed to be 
constant ®. For the atmospheric neutrinos crossing only the Earth mantle but not the Earth 
core {6n ^ 33.17°), for instance, the expressions for P^^, Kp and A^^^, in the two-layer model 
are given by (see, e.g., [41]): 

P'^ = sm'(^^]sm'29'l, (18) 



v{v) 



1 

2 



2E 



(19) 



AX, = l+(e-^^^-^-l)cos2e- (20) 

Here AE'^ and 6'^ are the difference between the energies of the two matter-eigenstate 
neutrinos and the mixing angle (in matter) in the mantle, which coincide with Am\]^/ {2E) 
and in vacuum, and L is the distance the neutrino travels in the mantle, L = 27?^ cos 9n, 
where i?0 = 6371 km is the Earth radius. Similar expressions have been derived in the case 
of neutrinos crossing the Earth core [36,37] (see also [41]). 

For a given neutrino trajectory in the Earth, the 2-ncutrino transition probability P^^ 
is a rather slowly varying function of the neutrino energy in the interval of interest E ~ 
(1 — 10) GeV. In contrast, the interference term Re(e~"^P74^?^,) in the expression of the 
{pfj) survival probability P^^ is a relatively fast oscillating function of E. As a consequence, 
the integration over the atmospheric neutrino energy in the indicated interval docs not affect 
significantly the probability P^*', while it tends to suppress strongly the effects due to the 
term Re{e~'^"'p A1^^,) . The specific properties of the oscillating term Re{e~''''^p AI^j^,) , present 
in the expression of the (i?^) survival probabihty P^^, have been discussed in the case 
of neutrinos crossing only the Earth mantle in [54]. A further discussion of the interplay 
between the term oc P^^ and that oc Re(A^) in the (P^) survival probability is given in 
Section 4. 

For Am|^ > 0, we can have P2u = 1 in the case of atmospheric neutrinos crossing only 
the mantle for neutrino energy and path-length given by Eres — 6.6 Am3^[10^'^ cV^] cos 2^13 
(7V"^"^«"[iV^cm-3])-i GeV and L[10^km] = (0.8 iVg'"'^'^[iVAcm-3] tan 2^13) Taking Aml^ ^ 
2.0 X 10-^ eV^ and sin^ ^13 = 0.05 (0.025) one finds that E^es = 6.6 GeV, and that we can 
have P2^ ^ 1 only if L ^ 8000 (10000) km. 

In the case of atmospheric neutrinos crossing the Earth core, it is possible to have P^*^ = 1 
for sin^ ^13 < 0.05 and Amg^ > 0, only due to the effect of maximal constructive interference 
between the amplitudes of the the z/g — > z/^ transitions in the Earth mantle and in the Earth 
core [36-39] The mantle-core enhancement effect is caused by the existence (for a given 

^Numerical calculations have shown [36,62] that, e.g., the 2-neutrino oscillation probability P^p of inter- 
est, calculated within the two-layer model of the Earth with iV™"" and for a given neutrino trajectory 
determined using the PREM (or the Staccy [60]) model, reproduces with a rather high precision the corre- 
sponding probability, calculated by solving numerically the relevant system of evolution equations with the 
more sophisticated Earth density profile of the PREM (or Stacey) model. 

^Thc effect differs from the MSW one [36] and the enhancement happens in the case of interest at a value 
of the energy between the resonance energies corresponding to the density in the mantle and that of the 
core. 
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i^-trajectory through the Earth core) of points of resonance-like total neutrino conversion, 
P'^" = 1, in the corresponding space of i/-oscillation parameters [38,39]. The points where 
P^*^ = 1 are determined by two conditions [38,39]. A rather complete set of values of Am'^^/E 
and sin^ 29i3 for which both conditions hold and P^" = 1 was found in [39] . The location of 
these points determines the regions where P^'^ is large, P^'^ ^ 0.5. For sin^ ^13 < 0.05, there 
are two sets of values of Am|i and sin^ ^13 for which P^*^ = 1: for, e.g., 9n = 0; 13°; 23°, we 
have P"^" = 1 at 1) sin^ 2^13 = 0.034; 0.039; 0.051, and at 2) sin^ 2^13 = 0.15; 0.17; 0.22 
(see Tabic 2 in [39]). For Am^^ = 2.0 x 10"^ eV^, P^^ = 1 occurs in the case 1) at 
E = (2.8-3.1) GeV. The effects of the mantle-core enhancement of P^'' (or P^^ if Am^^ < 0) 
increase rapidly with sin^ 26'i3 as long as sin^ 26'i3 ^ 0.06, and should exhibit a rather weak 
dependence on sin^ 2^13 for 0.06 ^ sin^ 26*13 < 0.19. 

In our further analysis amphtudes of interest A'f.^, Af^, etc are obtained by solving the 
evolution equation numerically using a realistic Earth density profile [59]. To simplify the 
notation we introduce the abbreviation Am^ = Aml^. 

3 Description of the Analysis 

Detection cross section. In the following we assume detection of atmospheric neutrinos 
of flavour a = e, /i by the charged-current interaction Ua+N — > a+X, where is the detector 
nucleon, a is the charged lepton, and X contains all the additional particles in the final state. 
Since we arc interested in energies large compared to the muon mass the total cross section 
for this reaction is equal for z/g and z/^, and we denote it by a^{E^), where p = viv) for 
neutrinos (antineutrinos). We take into account that in realistic detectors there will be an 
energy threshold Pthr for the detection of the charged leptons {e.g., muons). Therefore, the 
relevant cross section is given by 

where a^^^ is the total cross section for i^thr = 0, and = Q/SQ, = Q/3Q with Q ^ 0.44, 
Q ?a 0.06 being the average fractions of the momentum of the proton carried by quarks 
and anti-quarks, respectively. These numbers hold for a iso-scalar target, and our results 
do hardly depend on the precise values adopted for Q and Q. In our calculations we use a 

threshold of En^j. = 2 GeV. We assume that the detector provides enough information on 
momentum and energy of the final states, such that the initial neutrino energy and neutrino 
direction can be reconstructed with an accuracy of as and adh, respectively. 

Atmospheric neutrino fluxes. For the initial fluxes of atmospheric neutrinos we use 
the results of 3-dimensional calculations [63]. Since we consider only upward-going events 
with 0.1 < cos^„ < 1, where ^„ is the Nadir angle of the neutrino, one can neglect the 
height distribution of the neutrino production in the atmosphere. Furthermore, we average 
the fluxes over the azimuthal angle, such that we finally obtain fluxes ^^{E^, cos 9n), with 
a = e, ^ and p = 

Calculation of the event rates. Using the fluxes, cross sections and oscillation proba- 
bilities discussed earlier, the predicted spectrum of a-like events as a function of the neutrino 
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energy E„ and the Nadir angle cos On can be calculated by 

(22) 

where a, P = e, fi, a ^ p. To take into account the uncertainty in the reconstruction of 
neutrino energy and direction we fold this spectrum with Gaussian resolution functions Re 
and -Rdir: 




C/(C0S e'J i?dir(cOS On, COS O'J SHE',, COS 9'^) . (23) 



We assume a constant relative energy resolution, i.e., aE oc Ei^. Furthermore, we adopt the 
approximation of an angular resolution adh being independent of the neutrino energy. In the 
resolution function i?dir we convert Udir into an uncertainty on cos 9n by (j(cos 9n) = Cdir sin 9n 
with cJdir in radiant. 

Departing from the spectrum given in Eq. ()23j) we calculate event numbers divided into 
N^^ bins in E^, between E^^^ and E^^"^, and into iV^™ bins in cos6'„ between cos6'™^'^ and 
cos^J^'^^: 

N^k = C f dE, [ d{cos9n)ZiE,,cos9n) (24) 
Jj Jk 

where the integration boundaries are given by the jth bin in neutrino energy and the kth 
bin in cos^„. The normalization constant C is determined by fixing the total number of 
neutrino and antineutrino events to A^*°* for specified values of the oscillation parameters 
a; = (0i3,^23,Am2): 

^bin ^bin 

= E E E ^U^) ■ (25) 

j=l k=l p=u,u 

In general we choose for the normalization the "true" parameter values a;*'''^'^, the same which 
are used to simulate the "data" (see below). For the calculation of our actual observables 
we take into account that in a realistic experiment charge identification will not be perfect. 
Assuming that the charge assignment is correct for a fraction /id of all events we obtain the 
actual observables in each bin for neutrinos and antineutrinos by 

R^jk = /id A/Jfc + (1 - /id) A/Jfc /^^x 
R% = fmN^k + i^-fm)N^k ^ ' 

Hence, 100% correct charge identification corresponds to /id = 1, and /id = 0.5 implies no 
information on the charge, since neutrino and antineutrino events are summed with equal 
weight. For simplicity we assume here that /id does neither depend on the energy nor on 
the direction. 

The x^-analysis and systematic errors. We investigate the potential to determine 
the mass hierarchy by performing a statistical analysis based on a x^-function. Following 
the standard approach to construct a x-function for future experiments we calculate "data" 
by adopting "true values" a;'^'^'^'^ for the oscillation parameters: 

D'k = R%{^'n ■ (27) 



For a given experiment this might be only a rough approximation, since in general the ability to identify 
the charge of a particle does depend on the energy and on the direction relative to the magnetic field. 



9 



index / 


systematic effect 


value 


1 


overall normalization 


20% 


2 


u/u ratio 


5% 


3 


u^/ue ratio of fluxes 


5% 


A 7 


cos On dependence of fluxes 


0/0 


8-11 


energy dependence of fluxes 


5% 


12 


uncertainty on /id 


5% 



Table 1: Systematic uncertainties included in the analysis. 



In the theoretical prediction we take into account several sources of systematic errors by 
introducing 12 pull variables ^ = (^i, . . . , ^12): 



(28) 



with appropriately defined "couplings" ttJ^;- In the systematic error treatment we follow 
closely the description given in the appendix of [64] , where also a definition of the vr^^ ^ 
can be found (see also [3]). The systematic effects included in our analysis are listed in 
Tab. n We consider a fully correlated overall normalization error from various sources like 
uncertainties in the atmospheric neutrino fluxes, the cross sections, the fiducial detector 
mass, or efficiencies. We adopt a conservative value of 20%. Furthermore, we take into 
account an uncertainty in the neutrino/antineutrino ratio (including fluxes as well as cross 
sections) and an error on the ratio of e-like to /i-like fluxes. In addition to these normalization 
errors we allow also for uncertainties in the shape of the neutrino fluxes by introducing a 
linear tilt in cos6'„ and E^, uncorrected between the four fluxes of i/g, J^e, ^'/x, i^/^- And finally 
we take into account the effect of an uncertainty in the charge identification fraction /m. 
We consider the values for the systematics given in Tab. ^ as conservative estimates. We 
will use these representative values to illustrate the impact of such kind of uncertainties on 
the mass hierarchy sensitivity by comparing the with and without including them. 

Allowing for rather small event numbers per bin we adopt a x^-definition based on Poisson 
statistics: 



mm 



j=l k=l p=u,v 




(29) 



Since we are going to consider up to A?"^™ x A^^^™ = 20 x 20 bins, and the total number 
of events is expected to be of order of a few hundred, the actual number of events per bin 
will be often less than 1. In a real experiment of that kind an un-binned likelihood analysis 
should be performed in order to extract most information from the available data. In the 
absence of real data, however, the procedure outlined here still describes the performance 
of an "average" experiment. 



4 Qualitative Discussion 



In this Section we give a qualitative discussion of the effects allowing to distinguish the 
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Figure 1: The difference between the /Lt-lifce event energy spectra corresponding to Am^ > (NH) and 
Am^ < (fH), AS'^, defined in Eq. H32|l . In the left panels terms A and B are displayed separately, in 
the right panels the total effect is shown. In the upper (lower) panels an energy resolution of 5% (15%) is 
taken into account. The Nadir angle is fixed to cos6'„ = 0.9 (no smearing included). Thin (thick) curves 
correspond to sin'^ 6*23 = 0.5 (0.7). 



two types of neutrino mass liierarchy under the condition of relatively precise measurement 
of the event spectrum. This considerations will allow to obtain some insight in the numerical 
results presented in the subsequent sections. 

Substituting Eqs. p5|) and (fT^ into Eq. (j^^ . one obtains for the predicted event spectrum 
(see, e.g., [42]) 



SI oc <l>X[l + (r^sin2^^23-l)P|l] , 



SI oc ^yp 



1 + sin^ ^23 ( - - sin^ 023 ) PI - - sin^ 2^23 (l - Re (^^3)) 



(30) 
(31) 



where = r^^E^, cos 9n) is defined in Eq. (fTTjl and we have suppressed the dependence on 
El, and cos6'„. 

In order to determine the type of the neutrino mass spectrum one can explore the differ- 
ence in the event energy spectra as predicted by the NH and IH. To illustrate the effect for 
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/x-like events, let us consider 

ASP (X sin^ 623 - sin^ ^23) AP^ + 1 sin^ 2^23ARe (A'i,) , (32) 

term A term B 

with AX = X(NH) — X(IH). In Fig. ^we show AS^, where in the left panels we display 
the two terms in Eq. ()32j] separately, whereas the right panels show the total effect, i.e., the 
sum of "term A" and "term B" . In this plot we take into account also the effect of the finite 
energy resolution, i.e., we average AS"^ according to Eq. (PHj) assuming aE = 5% (15%) for 
the upper (lower) panels. Note however, that we do not average over the Nadir angle, and 
we fix cos6n = 0.9. 

From the upper left panel we observe that term B provides a large signal with a fast 
oscillatory behaviour in neutrino energy. In contrast, term A is much less sensitive to the 
neutrino energy, in particular its sign does not change. This implies that the averaging im- 
plied by worse energy resolutions mainly affects term B, whereas term A is rather insensitive 
to it. This effect can be observed in the lower left panel, where the signal from term B is 
significantly reduced with respect to the upper panel, but the size of term A is similar. As 
visible in the right panels, the total signal is also strongly affected by the energy averaging, 
since for good resolutions term B dominates. We find a qualitatively similar behaviour also 
for other values of the Nadir angle, as well as for the distribution in cos 6^ for fixed energy, 
and for antineutrinos. 

Furthermore, we illustrate in Fig.2]the dependence of the hierarchy effect on 623. Accord- 
ing to Eq. (j32|l . for term A this dependence is controlled by the factor sin^ 923(1 / — sin^ ^23), 
where in the relevant range of energies ~ 2.6 — 4.5. Hence, term A increases with sin^ ^23, 
very similar to effects in e-like events (see, e.g., [41] for a detailed discussion). In contrast, 
term B depends on sin^ 26'23, and therefore it has a maximum for 6'23 = 7r/4. This behaviour 
of terms A and B is visible in the left panels of Fig. ^ The dependence of the total effect 
on ^23 emerges from a non-trivial interplay of the two terms, where the effect of averaging 
plays a crucial role to control the relative size of them. We will return to this discussion in 
Sec. El where we present the dependence of the hierarchy sensitivity as a function of sin^ 6'23. 

5 Energy and Angular Resolution, Number of Bins, Charge ID, 
and Systematics 

In this Section we discuss in some detail the impact of "experimental" parameters such as 
energy and angular resolution, the number of bins used in the analysis, charge identification, 
and systematical errors on the ability to identify the neutrino mass hierarchy. Since our aim 
in this section is to identify the impact of these parameters on the sensitivity we perform 
a simplified analysis. For most purposes we take into account only statistical errors, i.e. 
we fix all the pull parameters $, in Eq. ()29|) to zero. Furthermore, we compare NH and IH 
for fixed oscillation parameters. More precisely, we calculate data for the NH and the true 
values £c*™*' = = (6*13, 6*23, Am^) with Am^ > 0. Then the is calculated for the same 
oscillation parameters but for IH: 

Ax^(NH;IH) = x'(^''";^''') (33) 
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1 5 10 15 201 5 10 15 20 



angular resolution [degree] energy resolution [%] 

Figure 2: Ax^ between NH and IH per 100 events as defined in Eq. H33|l as a function of tlie angular 
resolution (left) and energy resolution (right). The oscillation parameters are fixed to sin^ 2^13 = 0.1, 
sin^ 6*23 = 0.5, lAm^l = 2.4 x 10^^ eV^, and we use 20 x 20 bins in the intervals 2 GeV 10 GeV and 

0.1 < cos^n < 1, statistical errors only, and 100% charge identification. 

with a;™ = (^^13, ^23, ~^'^^)- The total event number is normahzed to 100 according to 
Eq. pHjl . It follows from Eq. that for statistical errors only {i.e., $ = 0), the is 
proportional to the total number of events, and hence, our results can be easily scaled for 
any event number. The scaling in the presence of systematical errors is discussed at the end 
of this section. 

We are aware of the fact that for atmospheric neutrino experiments charge identification 
is very difficult for electrons. Therefore, most likely only data consisting of /x-like events 
will be available. Nevertheless, for comparison we show our results also for an e-like event 
sample of the same size and with the same characteristics as the /i-like data. This will allow 
us to highlight the advantages or disadvantages of using the fi-like events, and to discuss the 
different requirements on the experiments. 

Let us first explore the impact of energy and angular resolutions. In Fig. |21 we show the 
Ax^ between NH and IH as defined in Eq. fl33|) as a function of the energy resolution and the 
angular resolution. In agreement with [53] we observe that for /i-like events these resolutions 
have a dramatic impact on the hierarchy sensitivity. This can be understood from comparing 
upper and lower panels of Fig. [H Due to the fast oscillations of the signal, averaging has a 
rather large effect and leads to a very different final signal. The left panel of Fig. El shows 
that especially for very good angular resolutions (Xdir ^ 5° and energy resolutions aE — 5% 
a very high sensitivity to the mass hierarchy can be obtained. Even with only 100 events 
Ax^-values corresponding to 2 — 3cr can be achieved, and the sensitivity of /i-like events 
becomes even better than the one from e-like events. 

In contrast, one can see from Fig. |21 that the sensitivity from e-like events is only slightly 
affected by angular and energy resolutions. The reason for this is that for e-like events the 
effect does not show such a pronounced oscillatory shape as in the case of /i-like events, but 
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Figure 3: Ax^ between NH and IH per 100 events as defined in Eq. H33|) as a function of the number of 
energy bins in the interval 2 GeV < E,^ < 8 GeV. The oscillation parameters are fixed to sin^ 2013 = 0.1, 
sin^023 = 0.5, |Am^| = 2.4 x 10~^ eV^, and we use 10 bins in the interval 0.3 < cos6'„ < 1, ddir = 5°, 
statistical errors only, and 100% charge identification. 



is more similar to the "term A" shown in Fig. ^ (See [41] for a detailed discussion of the 
hierarchy effect in e-hke events in the context of a water- Cherencov detector.) This imphes 
that averaging has a much smaller impact. 

We conclude from Fig. |2 that taking into account information on the neutrino energy 
and direction significantly improves the sensitivity to the neutrino mass hierarchy of /^-like 
events. This result is confirmed by Fig. El where we show the Ax^(NH;IH) as a function 
of the number of bins in neutrino energy N^^'^. The value A^g™ = 1 corresponds to only a 
rate measurement, whereas increasing values of A^^"^ imply more spectral information. We 
observe from that figure that for /x-like events with good energy resolution of = 5% the 
spectral information significantly improves the sensitivity to the mass hierarchy. Moving 
from just a rate measurement to 20 bins in increases Ax^ roughly by a factor of 6. For an 
energy resolution of 15% the optimal Ax^ is already reached for approximately 8 bins, since 
for A^^™ > 8 the energy resolution becomes larger than the bin size, and hence finer binning 
cannot increase the information. However, also in this case appropriate binning increases 
Ax^ approximately by a factor of 3 compared to a pure rate measurement. Furthermore, one 
finds from Fig. El that e-like events are much less sensitive to the number of energy bins, and 
already for A^^" ~ 2 the optimal Ax"^ is obtained. This is in agreement with our previous 
observations in relation with Fig. that e-like events are much less affected by averaging 
over energy and direction. 

In Fig. 13 we show the sensitivity to the mass hierarchy as a function of the charge iden- 
tification. As expected a bad charge ID has a significant impact on the hierarchy sensitivity 
since, depending on the hierarchy, the resonant matter effect occurs either for neutrinos 
or antineutrinos, and hence, distinguishing neutrino from antineutrino events is crucial. A 
charge ID of 90% leads approximately to a reduction of the Ax^ of 30% compared to perfect 
charge ID. For no information on the charge (/id = 0.5) the Ax"^ is reduced roughly by one 
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Figure 4: Ax^ between NH and IH per 100 events as defined in Eq. as a function of the charge 
identification in percent, 100% x /id, see Eq. (|26|) . The oscillation parameters are fixed to sin^ 26i3 = 0.1, 
sin^ 6*23 = 0.5, lAm^l = 2.4 x lO^^ eV^, and we use 20 x 20 bins in the intervals 2 GeV < E„ < 10 GeV and 
0.1 < cos6'„ < 1 and statistical errors only. 

order of magnitude. This is one reason why in the case of a water Cherencov detector with 
no possibihty to distinguish neutrino from antineutrino events very large exposures of the 
order of several Mt yrs are necessary to explore the mass hierarchy effect [50]. 

Finally we investigate the impact of systematical errors and the scaling of the with 
respect to the total number of events. In the left panel of Fig. we show the Ax^ between 
NH and IH as a function of the total number of events according to Eq. ()25|) including 
systematical uncertainties. The thin solid lines show explicitly for /i-like events that in the 
case of no systematical errors scales linearly with the number of events. Hence, the 
deviation of the curves in the left panel of Fig. from straight lines with inclination 1 is 
the manifestation of the systematical errors. We conclude from this figure that — given fixed 
oscillation parameters as defined in the caption — roughly 4000 events are needed to obtain 
a 2a effect for the hierarchy identification using /i-like events with resolutions of aE = 15%, 
(Tdir = 15°. In contrast, using e-like events or high resolution /i-like events, a 2a sensitivity 
seems possible already with ~ 200 events. 

The impact of the systematics is shown in the right panel, where we display the fractional 
decrease in Ax^ by switching on systematical uncertainties. One can read off from that plot 
that for iV*°* = 100 the impact is rather small. For the high resolution /i-like sample 
with aE = 5% and a^ir = 5° the Ax^ is reduced only by 3% due to the systematics; in 
the other shown cases the decrease in Ax^ is < 12%. In this regime the is dominated 
by statistical errors. However, the impact of systematics clearly increases for larger event 
numbers, leading to a reduction of Ax^ up to a factor 2 for ~ 10^ events. In Fig. El all the 
systematical uncertainties listed in Tab.Qare included. We have verified that the uncertainty 
in the overall normalization has practically no impact, despite the rather large value of 20%. 
The most relevant systematics are the ones which affect the relative number of neutrino and 
antineutrino events. In particular, the error on the u/u ratio and the error on the charge ID 
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Figure 5: We show as a function of the total number of events the Ay^ between NH and IH as defined in 
Eq. (p?!^ including systematical uncertainties (left panel), and the ratio between the Ax^(NH;IH) including 
systematical uncertainties and with statistical errors only (right panel). The values for the systematical 
uncertainties are given in Tab. Q The thin solid lines in the left panel correspond to statistical errors 
only (shown only for /i-like events). The oscillation parameters are fixed to sin^ 26*13 — 0.1, sin'^ 6*23 = 0.5, 
lAm^l = 2.4x 10~3 eV^, and we use 20x 20 bins in the intervals 2 GeV < .B^, < 10 GeV and 0.1 < cos6l„ < 1, 
and a charge identification of 95%. 



parameter /id are important. Moreover, also the errors on the cos^„ and Ey shapes are 
relevant, since we take them to be uncorrelated between neutrino and antineutrino events. 
In the case of e-Iike events also the uncertainty on the Ve/ i^ti flux ratio contributes noticeable, 
whereas this error has practically no impact for /x-like events. 

Let us comment on the relatively small effect of systematics in case of the high resolution 
/i-like data, visible in the right panel of Fig. This follows from the fact that the discrimina- 
tion between NH and IH is based on a very characteristic signal (compare Fig. Q), consisting 
of pronounced structures in the and cos 6'„ distributions, which cannot be easily mimicked 
by the systematic effects. If these structures are washed out to some degree by the averaging 
implied by worse resolutions, the impact of the systematics is increased, since the effect of 
changing the hierarchy can be reduced by adjusting the initial fluxes. The same argument 
applies also in the case of e-like events. 



6 Results from the General Fit 



Before we are going to present the results of a full flt including all parameters, we deflne 
in Tab.Elthree benchmark setups which we will use in the following. We give in the table the 
experimental characteristics used in the simulation and the analysis. All our results in 
the following are normalized to 200 events for the "true" parameters values. A rough scaling 
of the results can be performed by using Fig. El The difference between the two /i-like event 
samples S^^^^ and is given by the adopted values for energy and angular resolutions. Setup 



16 



setup label 


Chigh c C 

Of, 


event type 

total number of events for x^^^'^ 
energy range 
Nadir angle range 
number of bins in E,y x cos 6n 
charge identification 
systematical errors 
energy resolution cte 
angular resolution adir 


/i-like /i-like e-like 
200 200 200 
2 GeV < < 10 GeV 
0.1 < cos^„ < 1 
20 X 20 20 X 20 20 x 20 
95% 95% 80% 
included according Tab. ^ 
5% 15% 15% 
5° 15° 15° 



Table 2: Definition of example setups. 



^high corresponds to a "high resolution" sample with as = 5% and adh = 5°. These are 
rather optimistic values which might be achievable only for a very limited number of events 
by applying appropriate cuts on the muon and on the hadronic event. We discuss setup S^^^^ 
to illustrate the potential of an "optimal" experiment. The values = 15% and cTdir = 15° 
adopted for setup S"^ are more realistic and correspond roughly to the numbers given in [52] 
referring to the MONOLITH detector. Let us add that in a realistic experiment the /i-like 
data will consist of events of very diverse quality. The accuracy with which neutrino energy 
and direction can be reconstructed depends strongly on the observed muon energy and on 
the details of the hadronic event. Therefore, our Setups A and B correspond to ideal cases 
of well defined energy and direction resolutions, which nevertheless serve as representative 
examples to illustrate the sensitivity. With setup we show also results for e-like data. 
Being aware of the fact that charge identification is difficult for electrons we adopt a value 
of 80%. Although such a number might still be over-optimistic, we include e-like data in 
our discussion since already with relatively small numbers of events a good sensitivity to the 
mass hierarchy can be obtained. Moreover, as we have demonstrated in the previous section, 
for e-like events the accuracies of neutrino energy and direction reconstruction are much less 
critical than in the case of /i-like data (see Fig. Ej). 

In the following we discuss the sensitivity of these three example data sets to the mass 
hierarchy as a function of the assumed true parameter values of ^13 and 623. We present 
results adopting three different treatments of the oscillation parameters. First we fit the 
"data" without any additional information on the oscillation parameters, i.e., we minimize 
x'^{x^™'^', x) of Eq. ()29|) with respect to x, where the sign of Am^ has opposite values in 
^true ^ ^g^gg ^j^g parameters have to be determined by the experiment itself. 

Second, we assume that some external information on the oscillation parameters is available 
when minimizing with respect to x. This information is included by adding a prior function 
Xprior to the of Eq. with 



' sin^ 2^0, - sin^ 2^?,*^^ \ ^ / sin^ 20i , - sin^ 29%'"' 
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Figure 6: Ax^ of the wrong hierarchy as a function of the true value of sin^ 20i3 for the three data 
samples defined in Tab.|2 each consisting of 200 events: /x-like data 5^ (left), /i-like data with high energy 
and angular resolution ^^'^h (middle), and e-like data S'e (right). The assumed true parameter values are 
I Am^l = 2.4x 10"'^ eV^, sin^ 6*23 = 0.5, and for the solid (dashed) lines the true hierarchy is normal (inverted). 
For the curves labeled "fixed" the oscillation parameters are fixed to the true values, for the curves labeled 
"prior" external information is included according to Eq. (|35|l , and for the curves labeled "free" no additional 
information is assumed. 



and we are using the following representative errors at la: 

a(Am2) = 0.1 1 AmY'''' , ^(sin^ 2^23) = 0.1 sin^ 2^|"" , ^(sin^ 2^13) = 0.02 . (35) 

These accuracies on |Am^| and sin^2^23 can be obtained at the MINOS long-baseline and 
Super-K atmospheric neutrino experiments, whereas the indicated value for cr(sin^ 26'i3) can 
be reached in the Double-Chooz reactor experiment [17]. Note that in the way the informa- 
tion on 6'23 and ^13 is included in Eq. ()34|) we take into account that v^, [v^ disappearance 
experiments meassure actually sin^ 26*13 (sin^ 26*23). In particular, this implies that we include 
no prior information on the octant of ^23- And third, we also show our results by fixing the 
oscillation parameters to the true values, i.e., without minimizing the with respect to x. 
This corresponds to the ideal case of infinite precision on |Am^|, ^23 and 6*13. 

The results of such analyzes are presented in Figs. |Bl[7|and|Hl In Fig.lHlwe show the Ax^ 
of the wrong hierarchy as a function of the true value of sin^ 26*13. Clearly, a sizable value of 
6*13 is required since the effect disappears for ^13 — 0. Furthermore, one observes from this 
figure that external information on the oscillation parameters improves the mass hierarchy 
sensitivity: Without any external constraints (curves labeled "free" ) the Ax^ is significantly 
reduced compared to fixed parameters. It is interesting to note that for the high resolution 
sample the modest precision implied by Eq. leads already to a sensitivity very close 
to the one for perfectly known parameters (compare the curves labeled "fixed" and "prior"). 
Also shown in Fig. IHl is the dependence of the hierarchy sensitivity on the type of the true 
hierarchy (compare solid and dashed curves). One observes that if external information on 
the oscillation parameters is available for /x-like data, the sensitivity is very similar for a true 
NH and IH. In contrast, for e-like data the sensitivity is always better for a true NH. 

In Fig. [7| we answer the question of how many events are needed to exclude the wrong 
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Figure 7: The number of events needed to exclude the wrong hierarchy at 2cr (Ax^ = 4) as a function 
of the true value of sin^ 20i3 for the three data samples defined in Tab. |21 /i-like data S*^, /x-like data 
with high energy and angular resolution 5'^'*'', and e-like data Se- The assumed true parameter values 
are |Arn^| = 2.4 x 10"'^ eV^, sin^ 6*23 = 0.5, and for the solid (dashed) lines the true hierarchy is normal 
(inverted). For the oscillation parameters external information is included according to Eq. I|35(l . 



hierarchy at 2a. Considering true values sin^ 2^13 ~ 0.1 we find that for /x-like data S"^ event 
numbers of order fewxlO^ are needed, whereas in case of high resolution data S^^^^ already 
~ 200 events are sufficient. For e-like data O{10^) events are required. For comparison, from 
Refs. [52,53] we roughly estimate 4 events per kton year in a MONOLITH/INO type detector. 
Hence, with the proposed detector mass of 30 kton and a 5 year exposure ~ 600 events are 
obtained. We conclude that for the 15% energy and 15° angular resolutions adopted for 5^ 
only a very poor sensitivity to the mass hierarchy will be reachable. Therefore, improving the 
resolutions beyond these values and/or significantly larger exposures seem to be necessary. 

Finally we discuss the sensitivity to the mass hierarchy as a function of the true value of 
sin^^^23- So far we always have assumed maximal mixing sin^6'23 = 0.5. As shown in Fig. |H1 
the ability to identify the true hierarchy depends quite significantly on the true value of 6*23, 
and in general the Ax^ of the wrong hierarchy increases for increasing sin^ ^23- For example, 
for the data sample and fixed oscillation parameters, Ax^ increases by a factor 3 if sin^ 6'23 
is moved from 0.5 to 0.6. However, we note from the left panel of Fig. |Hlthat for 5*^ and 
sin^ ^23 > 0.5 the uncertainty on the oscillation parameters has a strong impact on the final 
sensitivity. The reason is a degeneracy between the octants of 6'23: For a true ^23 > 7r/4 the 
data can be fitted better with the wrong hierarchy for ^23 ~ — ^23- Note that our prior 
function Eq. (jH^ is insensitive to the octant of ^23- We have checked that if in addition to 
the accuracies given in Eq. (jHSj) also the octant of 6*23 were known, results close to the case 
of fixed oscillation parameters could be obtained ^ 

For the high resolution data S^^^^ shown in the middle panel of Fig. |H1 we find that the 

^The possibility of a precision measurement of 623 in an atmospheric neutrino experiment with an iron 
magnetized detector has been studied recently in [55]. 
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Figure 8: Ax^ of the wrong hierarchy as a function of the true value of sin^ 6*23 for the three data 
samples defined in Tab. El each consisting of 200 events: /j,-like data (left), /i-like data with high energy 
and angular resolution ^^'^h (middle), and e-like data Se (right). The assumed true parameter values are 
sin^ 2^13 = 0.1 and Am? = 2.4 x 10~^ eV^ (the true hierarchy is normal). For the curves labeled "fixed" 
the oscillation parameters are fixed to the true values, for the curves labeled "prior" external information is 
included according to Eq. (|35|) . and for the curves labeled "free" no additional information is assumed. 

maximal sensitivity is reached around maximal mixing 623 ~ 7r/4. As illustrated in Fig.Q in 
this case the "term B" contributes significantly to the sensitivity, and the behaviour shown in 
Fig. IHl follows from an interplay of "term A" (which is increasing with sin^ ^23) and "term B" 
(which is proportional to sin^2^23 and hence has a maximum at sin^^23 = 0.5). In contrast, 
in the case of 5^ the "term B" is suppressed due to the averaging implied by the worse 
resolutions, and hence, the increase of the sensitivity follows from the sin^ 023-dependence 
of "term A" shown in Eq. (jHS))- Also for e-like events the effect is similar to "term A", 
and according to Eq. ()H0|1 it is proportional to (r^sin^^23 ~ !)• As expected, we observe 
in the right panel of Fig. |H1 an increase of the sensitivity with sin^6'23, in agreement with 
Refs. [41,50]. 

7 Conclusions 

In this work we have reconsidered the possibility to determine the type of the neutrino 
mass hierarchy by using atmospheric neutrinos in a detector capable to distinguish between 
neutrino and antineutrino events. For not too small values of the mixing angle 613, Earth 
matter effects will lead to different signals in such a detector for a normal or inverted neutrino 
mass hierarchy. Having in mind magnetized iron calorimeters like MINOS, INO, or perhaps 
ATLAS, we have performed a x^-analysis including realistic neutrino fluxes, detection cross 
sections, and various systematical uncertainties. We discuss how the performance depends 
on detector characteristics like energy and direction resolutions or charge miss-identification, 
and on the systematical uncertainties related to the atmospheric neutrino fiuxes. Moreover, 
we show how the mass hierarchy determination depends on the true values of ^13, ^23, as well 
as on the true hierarchy. We also compare the potential of /x-like events to a (hypothetical) 
data sample of e-like events with similar characteristics. 
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Focusing on the detection of muons, one of our main findings is that the abihty to 
reconstruct the energy and direction of the neutrino is crucial for the mass hierarchy deter- 
mination. Assuming sin^ 2^13 ~ 0.1 and the optimistic values for the neutrino energy and 
direction reconstruction accuracies of 5% and 5°, respectively, the mass hierarchy can be 
identified at the 2a C.L. already with roughly 200 events (sum of neutrino and antineutrino 
events, including oscillations). In contrast, for less ambitious resolutions of 15% and 15°, of 
the order of 5000 events are needed. These numbers are based on the detection of muons 
with a correct charge identification of 95% and an energy threshold of 2 GeV. For illustration 
we have also considered the potential of using e-like event samples. Assuming energy and 
direction resolutions of 15% and 15° and a correct charge identification of 80% we find that 
of order 1000 events are necessary for a 2a hierarchy determination. 

The reason for the relatively high sensitivity which can be achieved using high resolution 
^-like events comes from the fact that the difference in the signals for normal and inverted 
hierarchies shows a characteristic oscillatory behaviour in the neutrino energy, as well as 
in the Nadir angle. If the energy and direction reconstruction are sufficiently precise to 
resolve these structures, a statistical analysis using energy as well as directional information 
(ideally an un-binned likelihood analysis) provides very good sensitivity to the hierarchy. 
For worse energy and direction reconstruction, these oscillatory structures are averaged out, 
which leads to a much worse sensitivity. Our results imply that for resolutions of 15% for 
the neutrino energy and 15° for the neutrino direction, in a MONOLITH/INO-like detector 
exposures of the order of a few Mton years are required to obtain a reasonable hierarchy 
sensitivity. However, we stress again that for high quality event samples with more precise 
reconstruction of the neutrino energy and direction the required exposures are significantly 
smaller. 
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